A Tool for Text Comparison

Text reuse is commonplace in academia and the media. An efficient algorithm for automatically detecting and measuring similar/related texts would have applications in corpus linguistics, historical studies and natural language engineering. In an effort to explore the issue of text reuse, a tool, named Crouch, has been developed based on the TESAS system (Piao 2001) for comparing and measuring text similarity and derivation in sets of texts. Given a set of candidate source and derived texts, this tool maps related sentences between a pair of texts employing n-gram, stemming and synonym matching approaches. Crouch examines the textual similarity of individual pairs of texts, and also clusters pairs of texts in a collection of texts according to their similarity. The comparison is directional, in that the comparison from a derived text to its source generally produces a higher score than a comparison in the opposite direction. This presents the possibility of detecting the direction of text derivation. The tool displays its comparison of a given pair of texts in a graphical interface to help users to analyse the texts. Furthermore, as the tool is written in Java and fully supports Unicode, it can be applied to many languages. At Lancaster University, it is currently being used to help detect related English newspaper articles in 17th century newspapers.